Exploring the Variability 
in the Validity of SAT 
Scores and High School 
GPA for Predicting First- 
Year Coiiege Grades at 
Different Colleges and 
Universities 


Jennifer L. Kobrin & Brian F. Patterson 
The College Board 


CdllegeBoanl 

inspiiin^ minds' 


Background 


• The SAT is one of the most researched tests, with 
hundreds of published validity studies. 

• Vast majority of studies including sampies at 
multiple institutions examine the relationship of 
SAT scores and HSGPA with college 
performance in aggregate. 

• Studies show that the relationship of SAT scores 
and HSGPA with FYGPA varies at different 
institutions/different types of institutions. 
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Prior Research Exploring Variations in 
SAT Predictive Validity 


• Baird (1983) used college characteristics as predictor 
variables to predict the size of the simple and multiple 
correlations of SAT and HSGPA with FYGPA (validity 
coefficients). 

• A few studies have examined the validity of the SAT using 
a multilevel view: 

• Brown and Zwick (2006) - Validity of SAT and SAT Subject Tests 
for predicting FYGPA at Univ. of CA. 

• Culpepper and Davenport (2009) - Variability in differential 
prediction of FYGPA using SAT and HSGPA by racial/ethnic 
group. 

• Shen et al. (2010) - created factors of institutional characteristics 

and used these to predict variability in SAT validity after taking 
account of statistical artifacts. (^^CMiiegeBo^ 
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Purpose of the Current Study 


To demonstrate the utility of a 
multilevel model to understand the 
relationship of institutional 
characteristics to the validity of the 
SAT and HSGPA for predicting first- 
year college GPA (FYGPA). 
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Method 


• Sample consisted of about 150,000 students from 109 
colleges and universities across the U.S. 

• SAT scores obtained from 2006 college-bound seniors 
cohort; HSGPA self-reported on SAT Questionnaire. 

• First phase replicated Baird (1983) to identify variables to 
use in the multilevel modeling. 

(Results from this phase will not be presented today) 

• Second phase used Hierarchical Linear Modeling (HLM) 
to model the variability in the relationship of SAT scores 
and HSGPA with FYGPA across the 109 institutions. 
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HLM Specifics 


• Analyses followed a step-wise approach, 
beginning with null model (one-way random 
effects AN OVA) 

• Full maximum likelihood estimation used in all 
models. 

• Random effects included for all student-level 
predictors. 

• Student-level predictors (SAT, HSGPA) were 
centered within institution (group-mean centered) 
and institution-level predictors were grand-mean 
centered. 
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Model-Building Summary 


. Model 1 - Null Model (ICC = 0.109) 

• Model 2 - added student-level predictors 
(SAT/1000 and HSGPA) 

• Model 3 - added average SAT/1000 and average 
HSGPA as Level 2 predictors. 

• Models 4 and 5 - added Level 2 predictors one at 
a time according to approximate coefficients and 
t-values estimated by the HLM program. 
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Final Model 


Level 1 : 

FYGPAij = poj + Pij*(HSGPA)y + p2j*(SAT total /1 000)y + ry. 

Level 2: 

Poj = Yoo + Yoi (average SAT)j + Y02 (average HSGPA)j + Y03 
(private )j + Yo 4( small)j + Yos (lai'ge)j + Yoe (very large)j + Poj 

Pij = Y10 + Yii(average SAT)j + Y12 (average HSGPA)j + Y13 
(average financial aid) + Ym (% white) + 

P2j “ Y20 Y21 (average SAT)j + Y22 (average HSGPA)j + Y23 

(% submitting SAT)j + Y24 (small)j + Y25 (large)j + Y26 (very 
large)j + P2j. 
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Results: Final Estimation of Fixed Effects for 
Intercept (Average FYGPA) 


Variable 

Coefficient 

Robust S.E. 

Intercept 

2.963 

0.036 

Average SAT/1000 

0.912 

0.151 

Average HSGPA 

0.027 

0.129 

Private Institution 

0.105 

0.039 

Small Institution 

-0.073 

0.041 

Large Institution 

-0.009 

0.030 

Very large institution 

-0.031 

0.045 
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Results: Final Estimation of Fixed Effects for 
HSGPA Siope (Predictive Vaiidity of HSGPA) 


Variable 

Coefficient 

Robust S.E. 

HSGPA slope 

0.419 

0.008 

Average SAT/1000 

-0.298 

0.129 

Average HSGPA 

0.053 

0.091 

Average Financial Aid 

-0.000 

0.000 

Percent of White first-year 
students 

0.002 

0.000 
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Results: Final Estimation of Fixed Effects for 
SAT Siope (Predictive Validity of SAT) 


Variable 

Coefficient 

Robust S.E. 

SAT slope 

0.813 

0.032 

Average SAT/1000 

-1.190 

0.234 

Average HSGPA 

0.991 

0.183 

Percent submitting SAT 

0.005 

0.001 

Small institution 

0.177 

0.066 

Large institution 

-0.032 

0.046 

Very large institution 

-0.058 

0.055 
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Results: Final Estimation of Variance 
Components 


Random 

Effect 

Variance 

Component 

df 

Chi- 

square 

P 

Intercept 

0.016 

86 

6459.03 

< .001 

HSGPA slope 

0.005 

88 

484.08 

< .001 

SAT slope 

0.023 

86 

463.65 

< .001 

Student-Level 

0.358 
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Summary of Findings 


Predictive vaiidity of HSGPA (once SAT scores are 
considered): 

• Stronger at institutions with higher percentage of white 
freshmen. 

• Not as strong at institutions with higher mean SAT and at 
institutions with higher average financial aid package. 

Predictive vaiidity of SAT (once HSGPA is considered): 

• Stronger at institutions with higher mean HSGPA, at 
institutions with higher percentage submitting SAT 
scores, and at smaller institutions. 
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study Limitations 


• Student-level model included only HSGPA and 
SAT scores; future research may include student 
demographic variables (gender, ethnicity, SES). 

• Scores on the 3 SAT sections were combined. 

• The criterion variable (FYGPA) may not be 
completely comparable across institutions. 
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Thank You! 


• Researchers are encouraged to freely express their 
professional judgment. Therefore, points of view or 
opinions stated in College Board presentations do not 
necessarily represent official College Board position 
or policy. 

• Access this presentation online at 
http://professionals.collegeboard.com/data-reports- 
research/cb/presentations 

• Please forward any questions, comments, and 
suggestions to: 

Jennifer Kobrin at: jkobrin@collegeboard.org 
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